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ABSTRACT 

Somp 3,000 proposals submitted for Research Grants in 
i^ducation and Small Grants, two of the major topic areas in the 1973 
FiPld Initiated Studies (FIS) program of the National Institute of 
!!:ducation (NIE) , are analyzed in this report to identify research 
features which seemed to receive priority emphasis and to describe 
some of the methcdological features of these projects in order to 
focus questions about the potential for integrated, cumulative 
knowledge in education. The rc-nort contains selected characteristics 
of research from a considerably longer list in the original report 
which is the third in a related series of investigations in which the 
characteristics of federally funded social research are assessed, 
f^bjectives of the study for NIF were to develop an instrument to 
assess the methodological quality of the research component of 
proposals; describe the substantive issues of proposals; determine 
issues removed and retained at each step of the NI^ review process; 
relate proposed characteristics of the research component; and assess 
rj'- 'temat ic preferences in the NIF proposal review process in terms of 
the r<='vi«w process used by Minnesota Systems Research, which 
conducted the project. (Author/JH) 
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IN TRODUCTION 

Potentially I educational research Involves an unusually broad range of 
theoretical perspectives, units of analysis and types of variables due In 
part to the number of specific disciplines which are relevant. This variety 
poses critical problems for Inter! elating research results and providing 
for the cumulative development of knowledge In education. Analytic discussions 
of the state of the field are holpful in this regard,^ pointing out overlaps, 
contradictions and unanswered questions In pest research. But what Increment 
In knowledge might we expect from the current Investment In research which 
will be completed in the near future? In what directions are we likely 
to go? With what kind of emphasis and balance? What variables and topics 
are being examined and are these Indeed the areas most In need of research? 
To what extent does this research have methodological features which will 
maximize its potential for making a cumulative addition to systematic 
knowledge in education? Little empirical description of these trends and 
potentials Is available in current literature. 

The following discussion provides information about characteristics of a 
relatively large collection of recently proposed and funded educational re- 
search, the 1973 NIE Field Initiated Studies (FIS) program. While we make 
no claim that this collection of research is representative of all current 
educational research, we do feel that this program has certain features 
which may help provide insight into some of these questions. It was a 
program which distributed some $10 million in research support, an amount 
which is substantial enough to have a large potential Impact. Some three 
thousand proposals were submitted, largely on topics considered relevant 
by those attuned to this funding source. Professionals both within government 
and outside made selection judgments and funded nearly two-hundred projects. 
These were professionals generally in a position to be aware of educational 
research trends and needs. Our description, then, is intended to help 
Identify some of the features of this research which seemed to receive priority 
emphasis, and to describe some of the methodological features of these projects 
in order to focus questions about the potential for integrated, cumulative 
knowledge in education. This paper pre^ients only selected characteristics 




of mtdrch from a considerably longer list In our original report. This 
stu4y is the third of a related series of Investigations In which we have 
been Involved to assess characteristics of federally funded social research.^ 
Previous studies 1n this series have focused primarily on methodological 
features of ongoing and completed projects. In this study, only research 
proposals were examined. This means that some features of research could 
only be judged on whether they were 'planned' or 'likely' rather than 
actually accomplished. 

The particular objectives for the study for NIE were to: (a) develop 
an Instrument to assess the methodological qual1t;y of the research component 
of proposals; (b) describe the substantive Issues of proposals In appropriate 
categories; (c) determine Issues removed and retained at each step of the 
NIE review process; (d) relate proposer characteristics to preferences 
for substantive Issues and characteristics of the research component of 
proposals; and (e) assess systematic preferences In the NIE proposal review 
process In terms of the review process used by Minnesota Systems Research, 
Inc (MSRI). 

Our focus in this paper is on the 2,951 proposals submitted for Research 
Grants In Education and Small Grants, two of the major topic areas in the 
1973 FIS program. Excluded from this paper are proposals in the Selected 
Discipline areas, which were reviewed in somewhat different steps. The 
steps through which a successful proposal travelled are shown In Figure 1: 



FIGURE 1 

STEPS IN FIS 1973 FOR RESEARCH GRANTS IN EDUCATION AND SMALL GRANTS 

1 Request for proposals mailed to approximately 25,000 people 

2 Screening of proposals by in-house personnel 

3 Panel review of proposals by outside professionals 

4 Funding decision based on panel recommendations 



As can be seen from the figure, the NIE review process consisted of screeners 
and panels. Practically all proposals approved by the panels were funded 
by the Institute. 



Blocks of variables examined In the course of the study were: 

1. Descriptive Methodology 

2. Characteristics of Principal Investigators 

3. Descriptive Content 

4. Evaluative Methodology 

These were measured by our two Instruments and these blocks were the basis 
of models analyzed In the larger study. Here, we will present selected 
outcomes In terms of these blocks of variables. 



METHODOLOGY 

The evaluation study design called for a set of research professionals to 
rate a stratified, non-proportional, random sample of proposals from the 
population. The population and sample are shown in Table 1. Two instruments 
were constructed for use by the raters to rate the proposals. One instrument 
dealt with a conceptual analysis of research proposals, including proposer 
characteristics and descriptive content variables. The other instrument 
dealt with research components and was oriented toward descriptive and 
evaluative methodological variables. A substantial amount of information 
was, thus, available (over 500 variables) in the data base. 

Proposals were randomly assigned to five raters who had been hired as summer 
Interns by NIE. All raters had relatively strong backgrounds in research 
methods and experience in doing research. Three of the raters were in 
sociology and two in human development and all had some professional interest 
in the field of education. All were either in doctoral programs or had 
relatively recently received their PhD degree. The raters received training 
before rating proposals and they were instructed to work Independently 
on the rating task. They were located at NIE in Washington, X. A rater 



TABLE 1 - NUMBER OF PROJECTS IN THE POPULATION AND 

STUDY SAMPLE BY DECISION STEP IN PROCESSING 
PROPOSED GRANTS FOR RESEARCH IN EDUCATION AND 
SMALL GRANTS SUBMITTED FOR NIE FIS 1973 



Number of 
Projects In: 


Rejected 
by 

Screeners 


Rejected 

by 
Panel 


Funded 


Total 
Proposals 


FIS 1973 
Population 


2345 


458 


148 


2951 


Study 
Sample 


220 


7o 


148 


441 



stability and consistency cncck wa:; .Je'i icjoeti Along with the main ratinq 
task.^ Raters were debriefed ai the end of the rating task as a further 
means of assessing rating nonn- anci c'nf initnns. They 'ere not privy 
to design details of our study until dftor .^11 ratiny hid been completed. 



CHARACTERIST ICS OF F UNDED flROPOS A L S 

Table 2 presents the distribution of selected characteristics of research 
proposals as measured by our instruments. These are broken down into four 
general categories corresponding '.o the blocks of variables shown in the 
model above. The left-most column in Table 2 is the distribution among 
all submitted proposals based on the weighted sample. The right-most column 
is the distribution among proposals ultimately funded. The center two columns 
express the net effect (difference between hose reviewed and those ap,)roved) 
of in-house staff screeners and of the outside, professional panelists. 

Over half (54%) of the funded research (and 57'r, of submitted proposals) 
was proposed by principal investigators who were in the field of education. 
Psychologists were second most frequent (36%) and about double their percentage 
representation among submitted proposals. Approximately five percent 
of the principal investigators were from minority groups as best this 
could be judged from limited available information. In terms of substantive 
topics, the area of development was most frequent among funded projects (55?0» 
with research on cognitive processes (45") and achievement (45%) close 
behind. Ability/aptitude, development and cognitive processes were sub- 
stantive topics which increased their percentage appearance by at least 
ten percent in the funded compared to the submitted proposals. 

Among funded projects, about half investigated some identifiable educational 
system (53%), a considerable drop from the emphasis found among all submitted 
proposals (75%). For projects which had an identifiable educational system, 
we sought to characterize that system in a number cf ways. One way involved 
a series of pairs of opoosed characteristics, such as 'established' versus 
'new'. Among proposals where an unambiguous decision between the pairs 



TAIL! I - THE OlSTRllUTlON OF SCIICTEO CHMACTCRItTICS OP MSEAMH MtOPOSA<,S_ ^ 
SroK St ARM SCREENING AM PMIEt DECISIONS. NIE»$ FIS fWOMN \%7i^ 



f^. PROPOSER CHARACTE R ISTIC S 

OUclpllne of Principdl Investigator 

Education 

Psychology 

Soc1o109y 
Minority Invtstlgitor 

t. DCSCRlPTlvr. CONTENT VARIABLES 

Subttintlvft Topics 

Instrvcclon 

Achl^vtmtnt 

Ability and Aptitude 

Development 

Cognitive Processes 

Attitudes end Values 

Classroom Role Structure 

Pf'eparatlon for Work 

Equality of Educational Opportunity 

Social Problems and Education 
Percentage of Projects Proposing/ Researching Educational 
System of Some Type 

Paired Characteristics of Educational System Proi»set/ 
Researched*** , . 

Educational Goals Instrumental (rather than Exprwive} 

Education as a Transmitter of Existing Values L Skills 

(rather than an Agent of Social Change) 

Academic (rather than Vocational) 

SoclalUatlon for Conformity (rather than Autonorny) 

Established (rather than New) 
Schoo^ Philosophy Typology**** 

Free Schools 

Open Schools 

Traditional 
Variables Used in the Research 

Effectiveness 

Behaviors 

Race-Ethn1c1'.y 

Growth and Development 

Creativity 

C. DESCRIPTIVE WKTHODOLOGICAL VARIABLES 

Classification of Research 
Cross Sectional Study 
Sample Survey 

Complex Analysis Techniques 
Qualitative Research 
Case Study 
Longitudinal Study 
Data Source 

New Questionnaire or Te^t Data 
New Observational Data 

p. EV A LUATIVE ME T HODOLOGICAL VARIABLES 

Major Strengths 
Applied 
Methods 
Theory 

Judged Likely to Meet Objectives 
Oats Judged to be Accessible 
Systematic Analysis rianned 
Methods Scales [Mean, (Standard Deviation)] 
Design (Range 0-5) 

Measurement Validity (Range 0-5) 
Sampling (Range 0-5) 

Statistical Procedures(Range 0-1) 



Oriol nal 
Population 
of Proposals 
Submitted 

56. 6« 
1B.S 
6.1 
6.3 



Care in Analysis 



(Range 0- 



42.8 
39.4 
30.4 
29.3 
26.7 
24.4 
21 1 
12.3 
10.8 
S.9 

74.8 



87.7 

81.6 
77.6 
58.1 
35.0 

69.5 
15.8 
14.7 

32.6 
25.8 
15.e 
15.0 
4.2 



41.9 

27.8 
24.4 
23.5 
16.7* 
10.0 

56.5 
26.5 



46.8 

21.3 
18.5 
47.7 
90.7 
74.1 



1.87 
.72 



(1.17 
(1.13 
1.37 1.12 
.19 ( .24 
.14 ( .25) 



Difftrcnce between 
Distribution Apwng ^ 
Proposals Soon By 
and Accepted By:** 
In^HOMSe Outside 
^creeners Panels 



If 



- e.i 

♦15.3 
♦ 2.3 

- .8 



♦ 7.1 

♦ 9.0 
♦22.5 

♦ 3.1 
. .4 

♦ B.6 



5.5 
9.1 



- 3.6 
♦17.5! 
♦13.4 
♦12.5 

- 3.6 
♦15.8 



♦ 3.1 

♦ 2.7 

♦ .4 

♦ .2 



-15.7 


A 7 a 


- 2.4 


* ft ^ 

▼ p. 4 


^ C A 


* K t 


♦ 7.0 


▼1 / .0 




4> S 2 


- ,.7 


♦15.7 


♦ .B 


♦ 4.5 


- 5.0 


♦ 1.5 


♦ 1.2 


- .5 


♦ 1.7 


♦ 2.5 


- 8.0 


-13.4 


- 6.4 


♦ 7.9 


♦ 4.3 


♦ 7.0 


♦ 4.5 


♦14.2 


- 5.3 


♦18.2 


- 2.0 


♦33.6 


♦ 8.7 


- 3.2 


- 8.6 


♦ 8.3 


- .1 


- 5.2 


- 7.5 


♦ 1.3 


♦ 3.1 


♦13.0 


- .6 


♦13.2 


♦ 5.3 


♦ 3.3 


♦ 1.4 


- 3.6 



♦ 7.8 
t 3.7 
♦13.9 

♦ 5.8 

♦ 4.0 
♦11.1 

♦ 1.5 

♦ .2 



♦ 2.6 
♦12.9 

♦ 9.3 
- 4.1 

♦ 6.9 

♦ 2.7 



♦.70 ( 


, .001 


+ .29 1 


-.21 


♦ .44 


,♦.26! 


♦ .21 1 


-.07 


♦ .28 


+ .16 


+ .12 


>-.07) 


+ .13 


;+.o4 


♦ .09 1 


.00) 


♦ .05 


; .00' 


+ .02 


>.02) 



Funded 
Propouli 

$4.U 
36.5 

8.8 

4.7 



34.5 
45.3 
42.6 
54.7 
45.3 
32.4 
2IS.4 
8.8 
11.5 
10.1 

53.4 



89.2 

92.9 
96.3 
71.0 
66.6 

75.0 
15.6 
9.4 

26.4 
41.9 
28.4 
23.6 
2.0 



56.8 
40.5 
60.8 
32.4 
20.3 
29.7 

63.5 
35.8 



45.9 
51.4 
41.2 

56.1 
yl.O 
92.6 

2.86 ( .97) 

1 .37 1.32) 

1.77 (1.22 

.40 ( .29 

.21 .27 
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nhe population. N=2951 . includes all proposals submitted to Grants for Research In Education and Jj*" J^^tj^g^J^ Jj^^j^l^ 

lom^hat dlfferenrst^ps: for-olation esf.i,nat« are based on weighted sample proportions fro.n a sample of 441 proposals. 
•*A (♦) me-ins 9n incroa-,0 in perccntflQo irm the previous sttp in the revirw proccis; a (-) means a decrease. 

♦*.T».c t.ase of porcentaGC-, 1. t^e number of projects about vn,e identl f iobl. ^y^^™,:;"^^ iSeri'JL'ntiriar''"" 

of r..v.h w1r could 1." ..n.v,l.i iuo..sly chr.-.l...-J. Llimlnatcd am all cases wher... both could uc checkrd or where ttie rater was 
yruMc to dctomifio whiHi should be chvO'-.d. 

»***The b.r.c of the p<..-';cnl.i'rs »i the nunJ.'-r of projects which could be cla-.iified as belonging in the three type^. 
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was possible. Table 2 provides the percentage emphasizing one side of the 
pair over the other side, giving some notion of relative emphasis among 
researched educational systems. Fund id research of thi;- type appears to 
be relatively homogeneous in emphasizing 'establi'ihed' (67%) rather than 
'new', where goals appear to be 'instrumental' (89%) rather than 'expressive', 
a transmitter of 'existing values and skills' (93%) rather than an agent 
of 'social change', 'academic' (96%) rather than 'vocational', and aligned 
toward socialization for 'conformity' (71%) rather than 'autonomy'. More 
than a ten percent increase in these percentages over that among submitted 
proposals occurred in all of the above pairs except ' Instrumental /express 1 ve ' . 
The educational research community appears to be relatively agreed upon 
the type of system needing research attention. 

One car look at these pairs in a somewhat different way in order to shed 
some initial light upon areas of emphasis. Two pairs asked for a classifica- 
tion of the researched educational system (if there was one) in terms of 
whether the teacher's and student's role was 'active' or 'passive'. From 
this a three- fold typology exhausted all but eight of the potentially clas- 
sifiable cases. Free school (both active) was by far the most popular 
focus (75%) among funded projects and submitted projects (70%) with open 
schools (student not teacher active) and traditional schools (teacher 
not student active) lagging far henind. This overall emphasis may be due 
ir part to rater's definition of an 'active' role. 

In terns of variables used in research, Tab"ie 2 indicates that creativity 
received very little attention (two percent of funded proposals), behaviors 
received attention in 42% of the funded proposals and effectiveness, race/ 
ethnicity and growth/development received attention in about a quarter of 
the proposals each. Behaviors and race/ethnicity were considerably more frequent 
among funded than among all s'jbmitted proposals. 



Proposed research was more frequently cross-sectional than longitudinal 
(57% versus 30%), sample survey rather than case study (40% versus 20%). 
an*! involved complex analysis techniques (generally more elaborate than 



univariate statistics and simple crost. tabulations) rather than qualitative 
research (61^ versus 32/.) anio ig funded resof^rch. Only qualitative and 
case-study research did not i icrease substantially in porcentage between 
all proposals and funded proposals. Although proposals involving observational 
data were substantially more frequent among funded projects than submitted 
projects, it is clear that questionnaire and test data are still favorites 
(64% among funded projects) 



Finally, in terms of evaluative methodology, 40 to 50 per-.ent of funded 
proposals were rated to have major strengths in methods, in theory, and 
in applied areas, and, except for applied strengths, each of these percentages 
was up substantially (ten per'.ent or moru) from the pool of all submitted 
proposals. Among funded projects, slightly less than 60% were judged to 
be likely to meet stated goals, nearly all (93%) planned some systematic 
analysis, and for nearly all (94%) data were judged to be accessible for 
the study, fiystematic analysis seemed preferred by screeners and panelists 
and as a result, this lype of project was more frequent among funded than 
subJnitted proposals. In addition to the above ratings, proposals were 
judged on a relatively large number of specific methodological features 
which were summarized in terms of five scales: Design, Sampling, Measurement 
Validity, Statistical Prjcedures and Care in Analysis (see appended note 
on scaling for a brief explanation of each scale). All except the latter 
two range from a low of 0 to d high or 5 (the latter two range from 0 to 1 
because only absence or presence of the characteristic was judged as only 
pronosals were available). On the average, scale scores were higher among 
funded than submitted proposals. Apparently screeners and panelists were 

sensitive to educational research needs for improved conceptual clarity 

5 

as noted by Travers' because the design scale i.howed the greatest average 
increase and its standard deviation decreased among funded projects as 
compared to the other scales. In separate analysis we have found that the 
relative frequency of methodological problems or fla/^:> rises markedly to 
substantial levels at design scores below about 3.0. The average design 
score of 2.9 among funded projects is about at this turning point, a sub- 
stantively important average shift up from the average design score (1.9) 
among all submitted projects. 



Even though a project has ai; idoquale stiiniiinu or. design, there are several 
other questions about proj'?c.t^ which r.hould De addressed, such as: 
When these projects are coinpi «ted , ai e Ihjy ViKcly to !■ we addrossed key 
topics? Is their niethodo loijy likely to support cumulative knowledge? 
Is there sufficient variety and arc pioneering ^reas likely to have been 
supported? If not» why and what. step«; irinht one take now to assure a more 
desired crop of findinas in {.he next five or ten years? 



SELECTION EMPHASIS OF IN -HOUSE S^REENERS AND OUTSID E PANELISTS 
The last section characterized educational research iri terms of the apparent 
interests and emphasis among those proposing research and among those select 
few who would be supported in producing findings to appear in the next two 
or three years. Another way of assessing the direction of the field is 
to examine decisions of those who are likely to be most aware of educational 
research possibilities and needs, both inside a funding agency and experienced 
professionals from the field. If these two selection groups differ, one 
might expect that the in-house screeners would emphasize agency goals and 
tend to eliminate types of proposals involving problem-areas to which they 
had been alerted. Outside panelists, one might expect, would emphasize 
selection in terms of substantive topics and, perhaps, methodology. 

An inspection of Table 2, shows that in-house screeners tended to screen 
out 'instruction' as a topic of research (a change of ten percent or more is 
considered worthy of note in this brief presentation) and screen in research 
on cognitive processes and that done by psychologists. Otherwise, screeners 
emphasized methodological characteristics of proposals, favoring those with 
major strengths in methods and theory, those planning systematic analysis, 
those likely to meet stated objectives and those that propose complex analysis 
techniques. It appears that methodological sophistication was an important 
concern and this emphasis appears to correspond to improvements in analysis 
sophistication that Travers found among the pre-1973 crop of research findings 
an older theme which apparently holds promise of being continued. 
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Invited panelists shared in-house screener interest in methods strength 
and tended to favor such projects although the amount of emphasis on methods 
and on theoretical strength was less than that of scree.iers. On the other 
hand, panelists tended to make their impact felt most in substantive areas, 
throwing out research involving identifiable educational systems, and selecting 
in topics such as development and attitudes anri values. Where proposals Involved 
an educational system, panelist preference went toward systems which were 
classified as 'established' rather than 'new', 'academic' rather than 
'vocational', and involving socialization for 'conformity' rather than 
'autonoiny'. Research involving behaviors and race/ethnicity as variables 
was also selected more frequently. Perhaps because of apparent interest 
in behaviors and development, panelists tended to select in projects involving 
longitudinal designs. As was true of screeners, panelists tended to favor 
projects proposing more complex analytic techniques. 

In a number of instances, screeners and panelists seemed to have divergent 
views as can be seen where one gro'jp screens out projects with certain features 
and the other group screens them back in (or vice versa). For example, 
in-house screeners cut the percentage of projects involving instruction 
by 16% but panelists increased the percentage by seven. Opposite tendencies 
occurred for attitudes and values as a topic of study (panelists increased 
the appearance of this topic), panels tended to favor open schools and screeners 
favored free schools, but both tended not to favor traditional schools as 
captured by this typology. None of these latter changes was substantial, 
however. Finally, panelists seemed more willing than screeners to tolerate 
prcjects which were not judged likely to meet 'stated objectives, perhaps 
indicating a willingness to take greater risks or a higher priority on 
substantive topics of interest. 

The screening and panel processes had some effect on the distribution of 
characteristics finally funded---the former more methodological, perhaps, 
and the latter more substantive---dl though by and large relative importance 
of topics remained fairly stable, indicating basic agreement with the dis- 
tribution of topics originally submitted by principal investigators. 



CHARAC TERISTICS RELATED TO l.TPUNn OlS .' u:j..,u:o_i^]:s 

If Travers' assessment of --idijcoMonal ro< .><)>•• :n is correct, it: would appear 
that conceptual izdti OP. and desiqn ar•^ ..ort if) n.-ou nf aitention currently 
than analytic or stutisticcil -jOpiii ii:i.at. on. Hv 'ound tl-ut the latter appeared 
to be substantially improved in 1973 aiid ':u>- description of funded proposals 
for current research sugtjostii th.it -.his asptaii o+ research is being given 
continued attention. .Mow initi.T: concoptua; izaiirui and design be improved? 
While we have no unique solution to this proi)leiii, it is possible to identify 
in a relatively qross way, soiiie c:i)d>'aotf}r;<.tici '-lese proposals which 
are associated with higher or lower scores on our overall design scale. 

The analytic technique we used for doing this was Automatic Interaction 
Detection Analysis. This procedure attempts to explain as well as possible 
the variance on a dependent variable by applying one-way analysis of variance 
techniques using permutations of a set of categorical independent variables 
to subdivide the population of proposals into a set of mutually exclusive 
subgroups. The independent variables employed consist of a number of 
composite variables developed through cluster analytic techniques and 
measuring descriptive methodological and descriptive content of proposals, 
as well as selected academic disciplines of the principal investigators. 
Each of the 14 predictor variables is measured as a presence-absence dichotomy. 
The descriptive methodology clusters were: l)experiment or observation, 
2)survey research, 3)common analytic techniques, 4)compiex analytic techniques, 
5)qual itative research and 6) "other' methods. The descriptive content 
clusters were: 7)human development, 8)professional training/program/structure, 
9)micro-level social systems and 10)macro-level social and political problems. 
These clusters are briefly explained in the attached note on scales. 
The selected academic disciplines of the investigators were: 11 )education, 
12)psychology , 13)sociology and 14)"other" disciplines. 

The dependent variable for this analysis was the 20 item research design 
factor scale which we constructed by applying factor analytic scaling 
procedures to a larger set of Likert-type rating scales measuring various 
aspects of design. The scale includes a number of more methodological 



features of research but it also includes items such as use of prior research 
knowledge, discussion of germane literature, explanation of the problem, 
control of relevant variables and other items which would seem to reflect 
conceptual strength. 



The tree diagram in Table 3 presents the results of this analysis, showing 
the mean design score for various subgroups of proposals. Examination 
of the various interactions will be left to the reader, but it should be 
noted that the highest mean design score for submitted projects occurs among 
projects which a) propose complex analysis techniques, b) involve experiments 
or observation as techniques, c) are proposed by principal investigators 
who are psychologists. At the other extreme, projects with the lowest 
mean design score are those which; a) do not propose complex analytic 
techniques, b) use techniques other than experiment or observation, c) do 
not use survey techniques, d) nor some other defined method, e) do not 
use qualitative techniques. Although several other variables, notably 
descriptive content variables, were used in the splitting process, essen- 
tially methodological techniques and one category of discipline of principal 
investigator accounted for the variance in the design scale. 



CONCLUSIONS 

Our data show that the initial population of proposals submitted to NIE 

emphasized certain topics rather than others, and that the in-house screeners 

and the panels expressed to a certain degree their preferences and had some 

impact in terms of shaping the ultimate characteristics of the research 

which was finally funded. Further, both the screeners and the panels appeared 

to be concerned with the conceptualization of the design of research and 

were able to accomplish improvement in this characteristic of the research 
funded. 

Our objective in this paper was to point out characteristics of proposals 
which constitute a significant portion of a future cohort of research findings. 
There are probably many alternative views about the desirability and promise 



of this type of resGarcli. Finally we hope this doricription will help focus 
discussion of educational research ohjectives by policy-makers, researchers 
and practitioners. 



rooTNon:_s 



See, for example, Sam Sieber and David Wilder (editors), THE SCHOOL IN 
SOCIF.TY, New York, The Free Press, 1973; or Robert M W Travers, editor, 
SECOND HANDBOOK OF RESEARCH ON TEACHING, Chicago, Rand McNally & Company, 
1973. 

In his introductory coinments for the SECOND HANDBOOK OF RESEARCH ON TEACHING 
(Chicago, Rand McNally, 1973), editor Travers summarized the reactions 
of contributing authors to the various literatures they reviewed as follows: 
"In many cases, after reviewing the literature, the author made the decision 
that the material was such that he could not write a chapter bringing 
the findings together---the complaint being that the research consisted 
of a patchwork of unrelated items that neither fitted together nor yielded 
a useful set of general izations .. .although there has certainly been an 
improvement in the statistical design of studies over the last decade, a 
corresponding improvement in conceptual design is often not apparent... 
the heavy emphasis upon what is wrong with educational research. . .reflects 
the general level of inadequacy of much of the research that has been 
undertaken." (Preface, page vii.) 

FINAL REPORT, METHODOLOGICAL ADEQUACY OF FEDERAL R&D PROJECTS, by 
Minnesota Systems Research, Inc, supported, in part, by Research Grant 
SRS 22- P- 55936 from the Division of Research and Demons trat-ion Grants, 
Social and Rehabilitation Service, Department of Health, Education and 
Welfare, Washington, DC 20201. And ASSESSMENT OF METHODS TO EVALUATE 
THE SCIENTIFIC RIGOR OF SOCIAL RESEARCH: A STUDY OF SIX REHABILITATION 
RESEARCH PROJECTS, by Minnesota Systems Research, Inc, sponsored by the 
Rehabilitation Services Administration and the Office of Research and 
Demonstrations of the Social and Rehabilitation Service and by the Office 
of Research and Evaluation Planning cf the Assistant Secretary for Planning 
ana Evaluation of the Department of Health, Education and Welfare under 
Grant Number 22-P-55036. 

The stability-consistency analyses provided evidence that the raters made 
stable and consistent judgments. In addition, some comparisons could be 
made with independently made classifications of projects and these matches 
were very close. For technical details on measurement issues see NATIONAL 
INSTITUTE OF EDUCATION FIELD INITIATED STUDIES PROPOSAL REVIEW, 1973, 
Final Report Contract Number NE-C-00-3-0287 by Minnesota Systems Research, 
Inc, sponsored by the National Institute of Education, Washington, DC. 

Travers (op. cit.) 

J A Sonquist and J N Morgan, THE DETECTION OF INTERACTION EFFECTS, Survey 
Research Center Monograph ^35, Institute for Social Research, University 
of Michigan, Ann Arbor, 1954. 





APPENDIX 



A NOTE ON SCALING 

Most of the items shown in previous tables are single judgments, but some are 

scales. We were able to characterize both the substantive topics included 

in a study and its various methodological features in terms of a large 

number of separate items selected to cover as broadly as possible the various 

facets of educational research. These items were used to create three 

sets of scales which serve to 'locate' a project descriptively and methodologically 

in an overall fashion. The scales were developed using cluster, smallest 

space and factor analysis guided by prior conceptualization of content and 

method. These scales are briefly described below: 

DESCRIPTIVE CONTENT: A combination of multidimensional scaling techniques 
(smallest space, cluster and factor analysis) was used to create four 
nonmutually exclusive clusters from 21 dichotomous descriptive content 
variables. The resulting clusters, their component items and the criteria 
for coding a proposal as being included in a cluster are shown in Table 4. 

DESCRIPTIVE METHODOLOGY: Using the same techniques as used for descriptive 
content variables, 22 dichotomous measures of descriptive methodology were 
combined into six nonmutually exclusive descriptive methodological clusters. 
As above, the resulting clusters, their component items and the criteria 
for coding a proposal as being characterized by the cluster are shown 
in Table 4. 

EVALUATIVE METHODOLOGY 

The evaluative methodology scales were constructed through extensive factor 
analyses of 66 Likert-type rating scales measuring various facets of methodo- 
logical quality. After determining the factor structure of this set of 
measures, the scales were "purified" by deleting relatively poor items to maximize 
their internal consistency reliabilities. Considerable evidence of construct 
validity was found for the scales (and is discussed in depth in our final 
report).- These scales are described on page 18. The specific items used 
in measuring each are listed along with internal consistency reliabilities 
in Table 5. 
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Pes ign implies those activities v^/hich involve planning, rather than conduct 
of research or analysis of the data. The most critical features of 
research design include formulation of the problem, familiarity with 
prior relevant work, awareness of basic assumptions, definition of key 
concepts, statement of hypotheses, specification and measurement of 
key variables and control of other relevant variables. 

The measurement validity factor concerns only the reliability and validity 
of measures. Ics focus is narrow in that it involves neither opera- 
tional ization and quantification per se nor issues of validity surrounding 
representativeness of sampling. 

The sampling factor refers to the selection of one or more cases to study 
from some larger class of potential cases. This includes definition 
of the target population, appropriateness of the sampling unit and sample 
size, and representativeness of the sample. 

The use of statistical procedures factor concerns the analysis stage 

of research. Specifically, it includes a range of quantitative statistical 

procedures and techniques. Its limitations involve its exclusion 

of non-statistical qualitative analytic procedures, of some very ordinary 

quantitative analytic procedures (such as univariate description, percentag 

tabling and crosstabulation) , and of items centeriny around quality 

of analysis and interpretation of findings. 

The car e of analy sis factor includes items focusing on the accuracy, 
orderliness and thoroughness of the analysis and the accuracy and 
appropriateness of the conclusions. It does not include items focusing 
on the care of design, sampling or measurement. 



TABIE 5 • EVALUATIVE METHODOLOGY FACTORS AND THEIR COMPONENT VARIABLES 



Factor* 



Design 



Cronbach's 
Alpha 

.95 



Measurement .83 
Validity** 



Sampling** .79 



Statistical .82 
Procedures*** 



Care of .76 
Analysis*** 



Variables 

Description of main concepts 
Explanation of research design details 
Description of design in application 
Specification of dependent variables 
Specification of independent variables 
Specification of control variables 
Discussion of germane literature 
Use of available prior knowledge 
Specification of hypotheses 
Discussion of assumptions 
Manageability of problem 
Experimental design 

Provision for hypothesis generating experiences 

Use of existing measurement procedures 

Quantification of variables 

Provision for adequate research resources 

Design efficiency 

Explanation of the problem 

Fit of design to the problem 

Control of relevant variables 

Use of scaling procedures 
Pretest of procedures and instruments 
Handling of validity issues 
Handling of reliability issues 
Checks on validity of instrument?^ 

Definition of the population of interest 
Use of probability sampling procedures 
Appropriateness of sampling unit 
Appropriateness of sample s^ze 
Random selection of cases 

Computerization of the analysis 
Use of measures of association 
Use of inferential statistics 
Use of correlational analysis 
Use of multiple regression analysis 
Use of analysis of variance 
Use of statistical controls 
Use of factor analysis 

Accuracy, orderliness and care of analysis 
Logic of data-conclusions linkage 
Data support for conclusions 
Completeness of analysis of available data 
Examination for possible biases 



♦Factor scores were computed by giving unit weightings to each of the component items 
for each factor, by sufmiing the ratings, and by dividing by the number of component 
items. Scores on the first 3 factors have a potential range of 0-5» while scores on 
the last 2 factors can potentially range from 0-1. 

**The rating categories of the variables comprising these factors ranged from 1 (unac- 
ceptable) to 5 (far above average), with 0 indicating Insufficient information or no 
consideration being given to a variable. 

•**The variables comprising these factors were judged as present (1) or absent (0) in a 
given proposal . 



